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Avoiding  Terminations,  Single  Offer  Competition,  and 
Costly  Change  Orders  with  Fixed-Price  Contracts1 
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Abstract 

Fixed-price  contracts  offer  the  promise  of  controlling  costs  but  are  less  likely  to  succeed  when 
there  is  uncertainty  regarding  requirements.  While  these  broad  principles  are  uncontroversial, 
disagreement  rages  regarding  the  practical  question  of  how  widely  they  should  be  used.  This 
study  tests  a  variety  of  hypotheses  regarding  what  contract  characteristics  are  associated 
with  better  performance  under  fixed-price  contracts.  Here,  performance  is  measured  across 
three  dependent  variables:  (a)  the  Number  of  Offers  Received  for  competed  contracts,  (b) 
whether  the  contract  was  terminated,  and  (c)  the  extent  to  which  change-orders  raised  the 
contracts’  cost  ceiling.  The  study  team  has  created  a  Bayesian  network,  populated  by 
completed,  publicly  reported  DoD  contracts  from  FY2007  to  FY2013  to  address  this  research 
question. 

The  public  purpose  of  this  process  also  includes  facilitating  future  acquisition  research  on  a 
range  of  topics.  To  support  future  research,  all  analytical  data  and  codes  developed  and/or 
used  are  posted  on  the  CSISdefense  GitHub  (Sanders,  2015).  This  resource  addresses  two 
vexing  issues  that  bedevil  a  wider  use  of  the  Federal  Procurement  Data  System  (FPDS)  by 
academia,  government,  and  industry  researchers,  namely  (a)  the  data-selection  barrier  to 
using  the  FPDS  and  (b)  the  difficulty  of  deriving  performance  outputs  from  FPDS. 


1  The  Center  for  Strategic  and  International  Studies  (CSIS)  does  not  take  specific  policy  positions; 
accordingly,  all  views  expressed  in  this  presentation  should  be  understood  to  be  solely  those  of  the 
author(s). 
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Introduction 

The  current  preference  within  the  executive  branch  and  congressional  oversight 
committees  for  fixed-price  contracts  is  a  reaction  to  external  budget  pressures  and  cost 
overruns  associated  with  certain  high  profile  acquisition  programs  (Younnossi  et  al.,  2007). 2 
But  even  the  DoD  acknowledges  that  fixed-price  contracting  is  not  a  panacea  for  all  that  ails 
federal  acquisition.3  Discussions  on  the  role  of  fixed-price  contracting  in  acquisition  is  really 
a  discourse  on  requirements,  risk  allocation,  and  uncertainty.  Accordingly,  as  a  contracting 
vehicle,  fixed-price  contracting  will  find  more  utility  in  specific,  well  defined  scenarios  such  as 
in  events  where  requirements  and  likely  costs  are  well  understood.  More  so,  than  in 
contracting  situations  characterized  by  uncertainty.  This  issue  is  acknowledged  by  OMB 
which  has  issued  the  guidance  governing  such  acquisition  transactions  (Office  of 
Management  and  Budget,  2009). 

Under  what  circumstances  are  fixed-price  contracts  most  likely  to  succeed?4  This 
paper  addresses  that  question  by  examining  seven  years  of  DoD  contract  transactions 
available  through  the  Federal  Procurement  Data  System  (FPDS).  The  project  compares  the 
performance  results  of  fixed-price  and  cost-based  contracts  across  a  range  of 
characteristics  that  typically  would  be  known  to  the  contracting  officer  before  the  contract 
was  signed.  While  FPDS  does  not  directly  measure  performance,  it  does  capture  three 
variables  that  directly  measure  the  possible  drawbacks  of  fixed-price  contracts.  These  are 

•  Single  Offer  Competition, 

•  Contract  Terminations,  and 

•  Cost-Ceiling-Raising  Change  Orders. 

“Single-Offer  Competition”  can  be  indicative  that  the  pricing  vehicle  involves  too 
much  risk  on  the  vendor  side,  which  makes  bidding  on  the  contract  unattractive  to  potential 
vendors.  Tracking  “Contract  Terminations”  addresses  the  risk  of  outright  failure  because  a 


2  Cost  growth  in  defense  acquisition  is  driven  by  several  factors  that  include  schedule,  unrealistic 
estimates,  acquisition  strategies  and  funding  availability.  Studies  of  weapon  system  cost  growth, 
notably  by  researchers  indicate  that  across  board  the  average  adjusted  total  cost  growth  for  a 
completed  program  was  46%  over  the  study  period  of  30  years. 

3  On  numerous  occasions,  Frank  Kendall,  the  under  secretary  of  defense  for  Acquisition,  Technology, 
and  Logistics,  has  underscored  the  need  to  use  appropriate  contracting  vehicles  by  pointing  out  the 
problems  that  a  fixed-price  contract  caused  during  its  development  of  the  A-12  Avenger. 

4  The  concept  “fixed  price”  deserves  some  further  illustration.  A  fixed-price  contract  suggests  a  price 
that  is  not  subject  to  any  adjustment  on  the  basis  of  the  contractor’s  cost  experience  in  performing  the 
contract.  And  according  to  provisions  in  the  Federal  Acquisition  Regulation  (FAR),  a  firm-fixed-price 
contract  is  suitable  for  acquiring  commercial  items  (see  Parts  2  and  12)  or  for  acquiring  other  supplies 
or  services  on  the  basis  of  reasonably  definite  functional  or  detailed  specifications  (see  Part  1 1 )  when 
the  contracting  officer  can  establish  fair  and  reasonable  prices  at  the  outset.  But  as  is  succinctly 
noted  in  the  DoD’s  (2014)  Performance  of  the  Defense  Acquisition  System:  2014  Annual  Report  (pp. 
iii-vi),  “Prices  on  fixed-price  contracts  are  only  ‘fixed’  if  the  contractual  work  content  and  deliverables 
remain  fixed”;  such  contracts  can  be  (and  often  are)  easily  modified  to  handle  unexpected  technology 
gaps,  engineering  issues,  or  shifting  threats,  leading  to  cost  growth.  At  times  fixed-price  vehicles  can 
be  virtually  indistinguishable  from  cost  plus  vehicles,  as  was  the  case  with  the  Air  Force’s  canceled 
Expeditionary  Combat  Support  System  (ECSS).  This  reading  of  “fixed  price  contract”  by  the  DoD 
guides  our  understanding  of  fixed-price  contracts  in  the  framework  of  this  study. 
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vendor  has  taken  on  more  risk  than  they  can  handle.  And  finally,  “Cost-Ceiling-Raising 
Change  Orders”  demonstrates  that  the  government  customer  finds  the  present  contract 
structure  unsatisfactory  and  may  indicate  cost  overruns. 

The  study  team  analyzes  the  drivers  of  these  three  dependent  variables  using  a 
Bayesian  network  populated  with  all  unclassified5  completed  DoD  contracts  from  FY2007  to 
FY2013.  The  study  objective  is  to  determine  when  fixed-price  contracts  are  most  effective. 
The  team  drew  hypotheses  from  the  literature  regarding  under  what  circumstances  the 
fixed-price  mechanism  would  lead  to  better  results.  The  team  also  developed  control 
variables.  These  controls  divide  the  data  to  capture  factors  not  relating  to  pricing  mechanism 
that  may  lead  to  better  or  worse  performance. 

Addressing  the  question  of  when  fixed-price  contracts  are  most  effective  is  only  a 
portion  of  the  public  purpose  of  this  project.  The  data  that  undergird  this  investigation 
originate  from  FPDS  and  are  thus  open  source  and  pertinent  to  a  wide  range  of  government 
contracting  questions.  With  this  chosen  approach,  the  study  team  simultaneously  addresses 
two  of  the  most  vexing  issues  that  have  bedeviled  the  wider  and  more  effective  use  of  FPDS 
by  academic,  government,  and  industry  researchers:  the  high  barrier  to  entry  for  accessing 
complete  and  relevant  data  and  the  difficulty  of  deriving  performance  outputs  from  FPDS.6 
To  ensure  reproducibility  of  this  analysis  and  to  provide  a  starting  point  for  future  research, 
the  entirety  of  the  dataset  is  published  through  the  CSISdefense  GitHub  account  along  with 
the  typographies  and  analytical  code  that  the  study  team  used  to  create  the  statistical 
models  (Sanders,  2015). 

Background 

Since  the  inception  of  the  federal  acquisition  business,  issues  of  performance  have 
vexed  policy-makers.  In  the  last  years,  major  data  analytics  work  by  DoD  analysts  found 
scant  statistical  correlation  between  industry  profit  margins  and  program  performance  in  cost 
and  schedule.  For  example,  rigorous  analysis  of  data  from  hundreds  of  major  weapons 
programs  strongly  suggest  that  hitherto  fixed-price  contracting  does  not  always  achieve  the 
set  goals  (DoD,  2014).  As  far  back  as  1949,  John  Perry  Miller,  in  Pricing  of  Military 
Procurements,  exposed  the  lack  of  symmetry  between  incentives  and  contract  performance 
(as  cited  in  Williamson,  1967,  p.  218).  In  the  same  hue,  scholarly  works  from  the  early  1960s 
by  Frederick  M.  Scherer  (1964)  and  Peck  also  support  the  notion  that  there  is  a  discrepancy 
between  values  attached  to  incentives  and  performance  results  (Kaysen,  1963).  Speaking  to 
this  issue,  Williamson  (1967)  glumly  stated, 


My  analysis  of  these  relations  leads  me  to  conclude  that  neither  the 
manipulation  of  profit  incentives,  nor  the  monitoring  of  contract  progress  can 
be  expected,  in  any  dependable  sense,  to  yield  significant  improvements  in 
contract  performance  as  long  as  the  specification  of  the  task  remains 
unchanged. (pp.  217-218) 


Procurement  has  long  been  at  the  top  of  the  government’s  laundry  list  of  activities 
that  need  improvement.  However,  this  policy  focus  has  come  in  cycles — with  attendant  ups 


5  There  is  no  regulatory  mandate  to  report  classified  contracts;  as  a  result,  CSIS  assumes  that  they 
are  not  included  within  FPDS. 

6  For  a  more  detailed  remark  on  these  factors,  please  see  the  following  section  of  this  report. 
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and  downs  over  the  years.  Practically  every  administration  since  the  1970s  has  embarked 
on  some  form  of  acquisition  reform  or  at  least  paid  lip-service  to  the  process.  Over  150 
major  studies  devoted  to  the  field  of  acquisition  reform  have  been  produced  since  the  end  of 
the  Second  World  War  (Schwartz,  2014).  Despite  these  efforts,  according  to  Congressional 
Research  Service  analyst  Moshe  Schwartz,  DoD  development  contracts  since  1993  have 
experienced  a  median  of  32%  cost  growth — not  adjusted  for  inflation.  Since  1997,  31%  of 
major  defense  acquisition  programs  have  had  cost  growth  of  at  least  15%.  Schwartz 
observed  that  every  year  between  1996  and  2010,  the  Army  spent  more  than  $1  billion  on 
programs  that  were  ultimately  canceled. 

In  2009  and  2010,  these  problems  led  to  the  latest  round  of  reforms,  including  the 
Weapon  System  Acquisition  Reform  Act  and  Better  Buying  Power.  The  first  set  of  DoD 
Better  Buying  Power  initiatives  called  for  greater  use  of  fixed-price  incentive  fee  contracts 
when  appropriate  (Office  of  the  Under  Secretary  of  Defense  for  Acquisition,  Technology, 
and  Logistics  [OUSD(AT&L)j,  2010).  The  second  iteration  of  these  reforms  in  2012 
suggested  fixed  price  for  low-rate  initial  production  but  emphasized  contracting  officer 
discretion  in  choosing  the  right  pricing  mechanism  (OUSD[AT&Lj,  2012).  In  2013,  the 
annual  report  on  the  Performance  of  the  Defense  Acquisition  System  reinforced  this  change, 
finding  that  use  of  fixed-price  contract  types  were  not  themselves  correlated  with  cost 
savings.  Throughout  this  debate,  all  sides  agree  that  fixed-price  contracts  are  more 
appropriate  in  some  cases  than  in  others. 

Study  Dataset  and  Performance  Measures 

The  dataset  for  this  study  consists  of  DoD  contracts  reported  in  FPDS  that  were 
initially  signed  no  earlier  than  FY2007  and  completed  by  FY2013.  Notable  exclusions 
include  classified  contracts  not  mandated  to  be  reported  in  FPDS,  contracts  funded  but  not 
managed  by  the  DoD,  and  Defense  Commissary  Agency  contracts  which  have  not  been 
reported  in  recent  years.7  To  enable  comparisons,  the  dataset  is  not  limited  to  fixed-price 
contracts. 

Determining  when  contracts  are  completed  is  the  most  challenging  portion  of 
compiling  the  dataset.  Contracts  closed  out  or  terminated  by  the  end  of  FY2013  are  included 
even  if  their  current  completion  dates  run  into  the  next  fiscal  year.  However,  many  contracts 
in  FPDS  and  in  the  sample  are  never  marked  as  closed  out  or  terminated  in  the  Reason  for 
Modification  field.  In  these  cases,  completion  status  is  based  on  the  current  completion  date 
of  the  most  recent  transaction  in  FPDS.  This  method  could  accidentally  include  contracts 
that  have  not  reached  their  ultimate  conclusion  dates  and  are  merely  dormant.  However,  the 
FY2013  sample  end  date  means  that  any  such  contracts  would  have  to  be  inactive  for  an 
entire  fiscal  year,  which  is  remarkably  unlikely. 

FPDS  raw  data  is  available  in  bulk  from  USAspending.gov  starting  in  FY2000. 
However,  data  quality  steadily  improves  over  that  decade,  particularly  in  the  commonly 
referenced  fields  of  interest  to  this  study.  In  most  cases,  unlabeled  rates  topped  out  at  5%  to 
10%.  The  critical  exceptions  are  the  Base  and  All  Options  and  Base  and  Exercised  Options 
fields,  which  report  contract  ceilings.  Prior  to  FY2007,  these  fields  are  blank  for  the  majority 


7  These  exclusions  are  common  with  other  DUG  work.  See  DUG  methodology  for  more  details: 
http://csis.orq/proqram/methodoloqy 
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of  contracts.  When  that  field  is  not  available,  calculating  the  percentage  growth  rate  due  to 
cost-ceiling-raising  change  orders  is  impossible.  In  addition,  this  study  classifies  contract 
size  by  initial  ceiling  and  not  total  obligations  because  the  latter  figure  is  dependent  on 
contract  performance. 

Because  a  key  dependent  and  independent  variable  are  not  available  prior  to 
FY2007,  the  study  team  chose  to  set  FY2007  as  the  start  date  rather  than  risk  sample  bias 
by  including  only  those  earlier  contracts  which  were  properly  labeled.  This  restriction  poses 
a  significant  limitation  in  that  no  contracts  of  more  than  seven  years  in  duration  can  be 
included  and  five-year  contracts  are  only  in  the  study  period  if  they  started  by  October  1 , 
2007,  or  were  closed  out  early. 

The  dependent  variables,  as  well  as  the  other  study  variables,  are  available  through 
the  project's  GitHub  page  (Sanders,  2015).  The  entire  dataset  includes  nearly  six  million 
contracts,  and  as  a  result  the  study  team  will  also  provide  smaller  samples  as  a  more 
accessible  starting  point.  The  means  of  sampling  will  be  determined  by  consultation  with 
members  of  the  likely  user  community  at  the  NPS  conference  and  the  interviews  during  the 
refinement  phase  of  this  project. 

The  largest  sampling  challenge  is  that  approximately  two-thirds  of  the  contracts  have 
a  total  obligated  value  less  than  $25,000,  but  these  contracts  represent  only  about  3%  of  the 
obligations  for  the  period.  As  a  result,  when  this  paper  deals  with  samples  for  computation 
reasons,  the  sample  is  weighted  by  total  obligations  to  better  reflect  DoD  spending  patterns 
rather  than  merely  describing  the  numerous,  but  comparatively  insignificant  in  aggregate, 
small  contracts.  The  Bayesian  model  developed  later  in  this  paper  is  calculated  using  the 
complete  dataset  but  includes  a  variable  differentiating  by  initial  contract  ceiling,  which  is 
intentionally  constructed  to  ease  study  of  larger  contracts. 

This  approach  addresses  two  of  the  largest  obstacles  to  wider  use  of  FPDS  within 
the  government,  academia,  and  industry.  First,  the  two  official  portals,  USAspending.gov 
and  the  FPDS  web-tool,  both  perform  a  critical  service  in  giving  access  to  contract 
transaction  in  aggregate  or  detailed  form.  However,  for  many  researchers  the  relevant  unit 
of  analysis  is  contracts  and  not  transactions.  Both  websites  can  be  used  to  access  the  full 
records  of  individual  contracts,  but  due  to  data  inconsistencies  and  bulk  download 
restrictions,  they  are  not  well  suited  to  larger  sample  studies. 

The  limitations  of  the  data  in  raw  form  can  be  overcome  by  downloading  the 
complete  data  feeds  via  the  data  tab  of  USAspending.gov,  but  with  each  year  accounting  for 
multiple  gigabytes,  this  represents  a  high  barrier  to  entry  for  researchers  who  lack  the  tools 
or  training  for  large  dataset  work.  This  challenge  is  further  increased  by  the  often  arcane 
nature  of  the  data  fields  and  the  need  to  undertake  cleanup  and  refer  to  multiple  columns  to 
get  to  data  of  interest.  This  first  challenge  is  attested  to  by  the  regular  calls  the  study  team 
receives  from  other  researchers  seeking  to  use  FPDS. 

The  second  obstacle  is  that  FPDS  almost  exclusively  measures  contract  inputs  but 
not  performance  outputs.  Measures  of  contract  performance  do  exist  in  other  databases,  but 
they  are  largely  inaccessible  without,  at  very  least,  an  official  government  purpose  or  permit. 
This  project  takes  a  step  towards  overcoming  that  problem  through  three  dependent 
variables  referenced  earlier  in  this  section.  These  variables  were  chosen  due  to  their 
relevance  to  fixed-price  contracting  and  availability,  but  they  are  also  applicable  to  a  wider 
range  of  research  questions.  Due  to  their  importance  and  broader  reference,  this  paper 
discusses  each  of  the  three  in  detail  in  a  subsequent  section. 
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Single-Offer  Competition 

Whether  or  not  a  contract  is  competed  is  primarily  an  input  into  performance. 
However,  single-offer  competition  also  reveals  information  about  the  request  for  proposals. 

A  solicitation  that  only  has  a  single  respondent  indicates  some  combination  of  three  factors: 
thinness  in  the  underlying  market;  a  failure  to  notify  or  give  adequate  response  time  to 
potential  competitors;  or  a  contract  that  is  unappealing  to  vendors.  That  final  point  is  of 
interest  to  this  study.  Fixed-price  contracts  transfer  risk  to  vendors,  and  if  vendors  perceive 
greater  risk  than  the  government  is  willing  to  pay  for,  then  fewer  vendors  may  be  willing  to 
bid. 

Methodology 

Non-competed  contracts  are  not  included  in  this  analysis  because  the  choice  of 
whether  or  not  to  compete  is  based  on  factors  that  are  already  known  before  the  choice 
between  fixed-price  and  cost-based  contract  is  made.  For  example,  when  a  sole-source 
award  is  justified  based  on  there  being  “only  one  source,”  that  rationale  refers  to  the  total 
number  of  potential  vendors  and  not  the  number  of  interested  vendors.  As  a  result,  for  the 
graphs  on  single-offer  competition  and  for  analysis  at  the  end  of  this  piece,  sole-source 
awards  are  excluded.  While  multiple  variables  are  used  to  judge  whether  a  contract  has 
been  completed,  only  one,  the  Number  of  Offers  Received,  is  necessary  to  determine  how 
many  vendors  really  did  submit  an  offer. 

Whether  a  contract  is  competed  is  calculated  largely  using  the  same  approach  as  in 
prior  CSIS  studies  (“Methodology,”  n.d.).  This  method  emulates  the  official  DoD 
methodology  to  the  extent  possible  when  using  raw  data  downloads  rather  than  the  FPDS 
webtool.  In  the  vast  majority  of  cases,  competitive  status  is  classified  for  the  entirety  of  the 
contract  duration.  Thus  if  a  contract  had  a  duration  of  three  years  and  was  competed  in  the 
first  year,  it  qualifies  as  competed  for  the  entirety  of  the  duration.  This  also  extends  to  single¬ 
award  indefinite  delivery  contracts,  which  are  classified  based  on  whether  the  original 
vehicle  was  competed  rather  than  consistently  treated  as  only  receiving  an  offer  from  the 
single  awardee.  The  Number  of  Offers  Received  is  calculated  using  the  same  strategy. 

To  see  more  details  on  the  construction  of  the  number  of  offers,  visit 
Contract_Competition.md  on  the  project’s  CSISdefense/Fixed-Price  GitHub  page  (“Fixed- 
price/Contract_Competition.md.,”  2015).  Summary  charts  are  included  in  the  parallel  CSIS 
report  on  Competition,  which  is  also  in  the  conference  proceedings. 

Terminations 

Abruptly  ending  a  contract  through  termination  is  a  challenging  endeavor  for  the 
government.  The  proximate  cause  of  the  termination  may  not  be  vendor  performance,  but 
instead  a  drastic  change  in  government  needs,  the  failure  of  a  related  contract,  or  the 
cancellation  of  the  entire  program.  However,  in  all  three  cases  the  government  has  the 
option  of  simply  running  out  currently  exercised  options  and  stopping  further  payments. 

Thus,  even  if  the  source  of  the  failure  was  outside  the  contract,  a  termination  indicates  that 
the  contract  was  unable  to  adapt  to  changing  circumstances.  Critically,  the  greatest 
vulnerability  of  fixed-price  contracting  will  result  in  a  termination:  if  too  much  risk  is  placed  on 
the  vendor  and  they  outright  fail  beyond  the  point  at  which  adjusting  the  contract  could  turn 
things  around.  In  this  instance,  the  government  may  lose  any  resources  that  have  already 
been  invested  as  well  as  paying  a  significant  cost  in  time  to  start  the  project  over. 
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Methodology 

Contract  termination  is  determined  using  the  Reason  for  Modification  field  in  FPDS. 
A  contract  is  considered  to  be  terminated  if  it  has  at  least  one  modification  with  the  following 
values: 


•  “Terminate  for  Default  (complete  or  partial)” 

•  “Terminate  for  Cause” 

•  “Terminate  for  Convenience  (complete  or  partial)” 

•  “Legal  Contract  Cancellation” 

These  four  categories  and  the  “Close  Out”  category  are  used  to  mark  a  contract  as 
closed.  As  is  discussed  above,  many  contracts  well  past  their  current  completion  date  never 
have  a  transaction  marking  them  closed,  however,  a  termination  is  an  active  measure  that 
mandates  reporting  unlike  the  natural  end  of  a  contract  which  can  go  unremarked. 

The  four  different  values  of  contract  termination  provide  useful  granularity,  but  for 
aforementioned  reasons  even  a  partial  termination  for  convenience  indicates  that  something 
has  likely  gone  awry.  Thus,  given  the  already  low  number  of  terminations,  the  study  team 
treats  a  contract  as  either  terminated  or  it  is  not,  rather  than  subdividing  by  type. 

To  see  more  details  on  the  construction  of  the  number  of  offers,  visit 
Contract_Competition.md  on  the  project’s  CSISdefense/Fixed-Price  GitHub  page  (“Fixed- 
price/Contract_Competition.md.,”  2015). 

Cost-Ceiling-Raising  Change  Orders 

Change  orders  are  not  as  severe  an  indicator  of  trouble  as  terminations.  A  change 
order  might  result  from  a  contract  being  adapted  to  a  changing  environment  or  even  being 
adapted  to  further  take  advantage  of  a  successful  innovation.  Even  when  the  change  order 
indicates  a  mistake,  it  may  often  not  be  on  the  vendor  side.  Instead,  requirements  creep 
prompted  by  the  government  may  add  expensive  new  tasks  to  the  contract.  The  affordability 
of  fixed-price  contracting  comes  in  part  from  their  simplicity  and  inflexibility.  Thus,  when 
fixed-price  contracts  are  subjected  to  a  large  number  of  change  orders,  whether  prompted 
by  government  or  vendor  actions,  this  is  a  warning  sign  that  a  different  form  of  pricing  may 
have  been  more  affordable. 

Methodology 

Similar  to  contract  terminations,  change  orders  are  reported  in  the  Reason  for 
Modification  field.  There  are  two  values  that  this  study  counts  as  change  orders:  “Change 
Order”  and  “Definitize  Change  Order.”  For  the  remainder  of  this  report,  contracts  with  at 
least  one  change  order  are  called  Changed  Contracts. 

There  are  also  multiple  modifications  captured  in  FPDS  that  this  current  study  will  not 
investigate  as  change  orders.  These  include: 

•  Additional  Work  (new  agreement,  FAR  Part  6  applies) 

•  Supplemental  Agreement  for  work  within  scope 

•  Exercise  an  Option 

•  Definitize  Letter  Contract 

The  Number  of  Change  Orders  refers  to  the  number  of  FPDS  transactions  for  a 
given  contract  that  lists  one  of  the  two  change  order  categories  as  their  Reason  for 
Modification.  The  vast  majority  of  contracts  do  not  receive  change  orders,  but  changed 
contracts  are  still  far  more  common  than  terminations. 
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This  study  uses  changes  in  the  Base  and  All  Options  Value  Amount  as  a  way  of 
tracking  the  potential  cost  of  change  orders.  The  Base  and  All  Options  Value  Amount  refers 
to  the  ceiling  of  contract  costs  if  all  available  options  were  exercised.  The  alternative  ceiling 
measure,  Base  and  Exercised  Value  Amount,  is  not  used  because  contracts  are  often 
specified  such  that  the  bulk  of  the  eventually  executed  contract,  in  dollar  terms  are  treated 
as  options.  In  these  cases,  the  all-inclusive  value  provides  a  better  baseline  for  tracking 
growth. 

The  Obligated  Amount  refers  to  the  actual  amount  paid  to  vendors.  This  study  team 
does  not  use  this  value  for  this  analysis  because  spending  for  change  orders  is  not 
necessarily  front-loaded.  For  example,  a  change  to  a  contract  in  May  of  2010  could  easily 
result  in  payments  from  May  2010  through  August  2013. 

The  %  Growth  in  Base  and  All  Options  Value  Amount  from  Change  Orders  is 
calculated  as  follows: 


Base  &AII  Options  Value  Increases  from  Change  Order  Modifications 


%  Growth  = 


Base  &AU  Options  Value  Amount  for  Original,  Unmodified  Transaction 


To  see  more  details  on  the  construction  of  the  number  of  offers,  visit 
Contract_ChangeOrders.md  on  the  project’s  CSISdefenses/Fixed-Price  GitHub  page 
(“Fixed-price/Contract_Competition.md.,”  201 5). 

Bayesian  Network  Model  Building 

A  variety  of  statistical  techniques  are  appropriate  for  inferential  analysis  on  when 
fixed-price  contracts  are  most  likely  to  be  successful.  The  study  team  chose  a  Bayesian 
network  approach  for  three  key  reasons.  First,  this  approach  scales  well  to  large  datasets, 
such  as  the  nearly  six  million  defense  contracts  completed  between  FY2007  and  FY2013. 
Traditionally,  only  a  sample  of  such  data  would  be  available,  but  thanks  to  FPDS  and 
modern  computing,  it  is  possible  to  analyze  the  entire  population.  Second,  while  a  Bayesian 
network  approach  and  other  similar  techniques  can  be  used  for  prediction,  it  is  particularly 
well  suited  to  understanding  the  how  the  different  pieces  of  evidence  are  interrelated. 
Because  this  project  seeks  to  provide  a  starting  point  for  future  research,  enhancing 
understanding  of  the  model’s  causal  logic  is  more  important  than  creating  a  model  which 
optimizes  the  ability  to  predict  outcomes. 

Finally,  the  knowledge  engineering  process  used  with  Bayesian  models — building 
connections  between  evidence,  called  whitelists  and  blacklists,  and  the  subsequent  model 
queries — is  well  suited  to  CSIS’s  strength  in  accessing  acquisition  domain  experts  and  data 
scientists.  The  initial  model  in  this  paper  will  be  built  upon  over  time  in  two  ways.  First,  it  will 
be  expanded  from  examining  only  the  Number  of  Offers  Received  to  including  all  three 
dependent  variables.  Second,  it  will  be  refined  by  consultation  with  additional  external 
experts  and  listening  to  their  insight  about  which  pieces  of  evidence  should  be  connected 
and  where  more  granularity  may  be  appropriate. 

The  model  is  built  in  the  open  source  statistical  programming  language  R  using  two 
modules.  The  module  BnLearn  is  used  for  the  Bayesian  network  learning  process,  which 
turns  the  collected  data  into  a  directed  graph  that  is  acyclic,  which  is  to  say  there  are  no 
loops  (Scutari,  2010).  The  module  gRain  is  used  for  the  second  part  of  the  process,  creating 
the  conditional  probability  table  and  then  querying  the  resulting  multiples  (Hojsgaard,  2014). 
Both  modules  are  also  open  source  and  the  data  as  well  as  the  processing  and  analytic 
programming  code  used  to  implement  this  process  are  available  through  the  CSISdefense 
fixed  price  GitHub  repository  (Sanders,  2015). 
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Description  of  Evidence 

As  with  most  statistical  models,  the  first  step  with  a  Bayesian  network  is  to  gather, 
clean,  and  transform  the  data.  Each  piece  of  evidence  was  collected  by  first  applying  CSIS 
transaction-level  lookup  tables.  Once  the  transaction  level  data  was  categorized  in  SQL 
server,  it  was  then  collated  into  contract-wide  values.  The  first  step  in  this  process  was  done 
using  codebooks  available  at  the  lookup-tables  repository  of  the  CSISdefense  GitHub 
account.  The  combination  stage  was  done  on  a  variable  by  variable  basis  and  that  process 
is  covered  in  the  fixed-price  repository  of  the  CSISdefense  GitHub  account.  The  last  stage 
was  conversion  into  evidence  notes,  each  having  between  two  and  eight  distinct  states.  The 
processing  required  for  a  Bayesian  network  increases  exponentially  with  each  new  node  of 
evidence.  As  a  result,  the  initial  model  is  intentionally  minimalist,  and  our  study  team  will  add 
more  granularity  to  the  model  as  developments  warrant. 

Contract  Fundamentals 

These  nodes  of  evidence  are  largely  set  by  the  needs  of  the  relevant  portion  of  the 
Department  of  Defense  rather  than  chosen  by  the  acquisition  official.  There  are  choices  to 
be  made  within  them,  for  example,  whether  to  fill  a  need  directly  with  as  a  product  or  via  a 
service  provider.  However,  as  a  rule  these  nodes  of  evidence  influence  the  contracting 
method  and  not  vice  versa. 

Who  (Component:  Army/Navy/Air  Force/Other  DoD):  Determined  by  the 
contracting  office  rather  than  the  funding  office.  This  will  be  referred  to  as  “component” 
throughout  this  discussion. 

What  (Platform:  Air/Land/Vessel/Electronics  &  Communications/Missiles  and 
Space/Weapons  and  Ammunition/Facility  Related  Services  &  Construction/Other): 

Determined  by  the  combination  of  the  claimant  program  code  for  the  platform  when 
available  and  otherwise  via  the  product  or  service  code.  This  will  be  referred  to  as  “platform” 
throughout  this  discussion. 

PSR  (Product/Service/R&D):  Determined  by  the  product  or  service  codes,  with 
R&D  management  and  support  being  treated  as  a  service. 

Inti  (International:  Just  U.S./Some  International):  Based  on  the  place  of 
performance.  Those  contracts  with  any  transactions  in  foreign  countries  are  treated  as 
having  some  international.  Lookup  tables  are  primarily  used  when  imputing  data  is 
necessary  due  to  a  missing  or  malformed  value  in  either  the  country  or  U.S.  state  place  of 
performance  fields. 

Link  (#  of  linked  contracts:  none/1-749/750+):  This  calculated  column  is  the  study 
team’s  first  effort  to  account  for  the  possibility  of  problems  cascading  from  other  related 
contracts.  For  those  contracts  without  a  system  code,  the  value  is  set  equal  to  the  number  of 
preexisting  contracts  in  the  same  contracting  office  that  share  a  Platform  characterization 
(excluding  the  Facility-related  Services  and  Construction  [FRS&C]  and  other  platforms).  For 
those  contracts  with  a  system  equipment  code,  the  value  is  equal  to  the  number  of 
preexisting  contracts  sharing  the  system  equipment  code.  This  value  is  then  supplemented 
with  the  number  of  contracts  with  the  same  platform  and  contracting  office  (with  the 
aforementioned  exclusions)  that  are  not  labeled  with  any  system  or  equipment  code.  This 
field  will  be  referred  to  as  “Interlinkages”  throughout  this  discussion. 

See  the  fixed-price  repository  of  the  CSISdefense  GitHub  account  for  processing 
code  used  for  this  calculation. 
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Contract  Approach 

The  contract  approach  refers  to  those  contract  characteristics  that  are  chosen  by  the 
relevant  acquisition  officials  in  the  pursuit  of  a  successful  outcome. 

Comp  (Competition:  Comp./No  Comp.):  Is  determined  using  the  standard  CSIS 
methodology,  with  the  critical  exception  that  the  numbers  of  offers  received  is  treated  as  a 
separate  piece  of  evidence. 

Ceil  (Ceiling:  $15,000/$100,000/$1,000,000/$30,000,000):  Refers  to  the  initial 
ceiling  on  total  potential  contract  obligations.  Is  set  by  the  initial  Base  and  All  Options  Value 
for  the  contract.  This  value  was  chosen  rather  than  the  initial  Base  and  Exercised  Options 
Value  because  exercising  options  happens  regularly  during  the  course  of  an  on-time  and 
on-budget  contract. 

Dur  (Duration:  One  day  to  Two  Months/Seven  Months  to  a  Year/More  than  a 
Year):  Refers  to  the  duration  and  is  calculated  using  the  number  of  days  between  the  initial 
effective  date  and  the  current  completion  date  for  the  contract.  The  ultimate  completion  date 
is  also  available  but  was  regularly  unlabeled. 

FxCb  (Fixed  price  or  Cost-base):  Fixed  price  includes  all  forms  of  fixed-price 
contracting  except  fixed-price  level  of  effort.  That  comparatively  rare  form  has  been 
described  in  meetings  with  DoD  officials  as  exhibiting  more  properties  of  cost-based 
contracts.  Cost-base  includes  all  forms  of  cost-plus  contracts  as  well  as  time  and  materials 
and  labor-hours  contracts. 

Indefinite  Delivery  Vehicle  (IDV):  Indicates  whether  or  not  a  contract  is  one  of  the 
many  forms  of  Indefinite  Delivery  Vehicles  (IDVs).  This  is  a  contracting  approach  in  which  a 
single  root  contract  is  used  as  a  basis  for  multiple  other  contracts. 

Contract  Outcomes 

Offr  (Number  of  Offers  Received:  1,  2,  3-4,  5+):  This  is  the  first  dependent 
variable  and  is  described  in  greater  detail  earlier  in  this  document.  Cases  with  no 
competition  are  categorized  as  only  one  offer,  although  for  hypothesis  purposes,  the 
existence  of  competition  will  typically  be  included  in  the  model  query.  The  remaining 
dependent  variables  will  be  added  in  to  the  model  in  coming  research  stages. 

The  Whitelist  of  Mandatory  Arcs  Between  Pieces  of  Evidence 

The  whitelist  is  a  collection  of  directed  arcs  between  pieces  of  evidence  that  must  be 
included  in  the  final  model  whether  or  not  the  learning  algorithm  recommends  them. 
Developing  the  whitelist  was  an  iterative  process.  First  the  team  determined  which  pieces  of 
evidence  were  most  strongly  linked  and  then,  after  seeing  initial  results,  added  further 
connections  where  the  learning  algorithm  could  not  determine  the  flow  of  causality.  Finally, 
the  team  compared  different  versions  of  the  model  created  using  variant  algorithms  and 
added  those  arcs  which  the  study  team  deemed  important  but  that  were  absent  in  some 
models.  The  whitelist  in  Figure  1  is  the  final  one  at  the  time  this  paper  was  written,  including 
all  three  iterations  of  building  the  list. 
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Figure  1 .  Whitelist  of  Arcs  That  Must  Appear  in  the  Bayesian  Model 

In  interpreting  this  graph,  each  circle  is  a  piece  of  evidence  or  node.  Each  arrow  is  a 
directed  arc,  and  the  parent  node  influences  the  child  node  that  the  arrow  points  to.  For 
example,  there  is  an  arc  from  the  parent  Component  ("Who")  node  to  the  child  Platform 
("What")  node  because  the  different  components  buy  different  mixes  of  platforms.  Thus  if 
Component  is  “Army”  then  the  Platform  is  substantially  more  likely  to  be  “Land  Vehicles”  and 
less  likely  to  be  “Vessels.”  So  long  as  no  loops  are  formed,  each  node  can  be  linked  to 
multiple  or  no  other  nodes,  for  example  Product/Service/R&D  (PSR)  influences  both  “Fixed 
price  or  Cost-base”  and  Interlinkages  (“Link”)  and  “Number  of  Offers  Received”  (Offr)  has 
arcs  coming  from  both  comp  and  “Fixed  price  or  Cost-base.” 

Working  from  the  bottom,  “Number  of  Offers  Received”  has  “Fixed  price  or  Cost- 
base”  and  Comp  as  its  parents  because  the  number  of  offers  directly  depend  on  whether  a 
contract  is  competed  and  “Fixed  price  or  Cost-base”  is  the  study  variable,  and  thus  the 
connection  with  the  dependent  variable  is  highly  of  interest.  On  the  next  level  up, 
Interlinkages  is  a  constructed  variable  and  both  Platform  and  PSR  are  used  to  create  it. 

Who  also  plays  a  role,  but  primarily  at  the  contracting  office  level  and  thus  the  connection  is 
not  as  direct.  ’’Fixed  price  or  Cost-base’s”  arc  from  PSR  was  there  from  the  start,  as  R&D 
contracts  are  classically  the  domain  of  cost-based  pricing.  ’’Fixed  price  or  Cost-base’s” 
second  parent,  Ceiling,  was  added  near  the  end  of  the  process  because  “Fixed  price  or 
Cost-base”  only  had  one  parent  in  some  of  the  models  and  consistently  including  Ceiling  in 
the  rest.  Further  analysis  via  cross-sectional  graphs  showed  that  after  accounting  for  PSR, 
Ceiling  appeared  to  have  the  strongest  influence  on  “Fixed  price  or  Cost-base.”  Namely, 
contracts  with  high  ceilings  are  notably  more  likely  to  be  Cost-Based  or  Combination. 

Platform  is  influenced  by  Components  in  another  straight  forward  connection  as  was 
covered  in  an  earlier  example.  IDV  is  a  parent  of  Ceiling  because  the  learning  algorithm 
could  not  decide  on  the  direction  of  the  relationship.  The  study  team  decided  that  IDV  was 
the  parent  because  choosing  an  IDV  can  often  mean  choosing  to  break  a  goal  into  multiple 
smaller  pieces,  each  with  a  small  scope.  Thus  the  Ceiling  of  a  project  can  depend  on 
whether  or  not  the  contracting  officer  feels  an  IDV  is  available  and  appropriate.  The  last  two 
nodes,  Inti  and  Duration,  do  not  have  any  whitelist  entries  at  this  time.  During  model 
creation,  the  study  team  experimented  with  linking  Duration  to  “Number  of  Offers  Received,” 
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but  that  caused  the  learning  algorithm  to  reject  the  stronger  connection  between  Ceiling  and 
“Number  of  Offers  Received.”  The  team  could  force  both  connections,  but  it  is  not  necessary 
to  do  so  and  the  relationship  between  Duration  and  “Number  of  Offers  Received”  appeared 
complex  in  cross-sectional  graphs.  As  will  be  shown  later,  this  does  leave  Duration  without 
any  children,  but  this  problem  will  likely  be  remedied  once  additional  dependent  variables 
are  added. 

The  Blacklist  of  Mandatory  Arcs  Between  Pieces  of  Evidence 

Black  lists  are  the  inversion  of  white  lists:  arcs  that  may  never  be  included  in  the 
model  regardless  of  the  findings  of  the  learning  algorithm.  Developing  the  blacklist  was 
similarly  an  iterative  process,  although  with  one  notable  exception,  most  of  the  revisions 
were  merely  adding  more  arcs  to  the  blacklist  to  correct  for  possibilities  overlooked  in  prior 
iterations. 

For  Figure  2,  the  red  lines  indicate  arcs  that  are  not  allowed.  For  example  the  arrow 
from  “Number  of  Offers  Received”  to  Platform  means  that  the  number  of  offers  cannot  have 
a  causal  influence  on  what  sort  of  platform  is  being  bought.  In  many  cases,  this  is  for 
straightforward  causal  reasons.  The  evidence  regarding  contract  fundamental  can  influence 
the  contracting  approach,  but  not  vice  versa.  This  graph  is  has  many  more  arcs  than  the 
prior  Whitelist  graph  because  it  is  straightforward  for  experts  to  establish  which  evidence 
factors  are  decided  earlier  along  the  timeline  or  are  take  precedence  over  deciding  other 
related  factors.  For  example,  the  relevant  acquisition  official  will  typically  first  determine 
whether  they  can  compete  a  given  contract  and  only  then  determine  what  vehicle  or  pricing 
mechanism  would  be  appropriate. 

The  existence  of  a  blacklist  arrow  does  not  mandate  that  there  is  a  connection  going 
the  other  direction.  In  fact,  that  is  the  point  of  the  blacklist,  to  prevent  spurious  connections 
from  being  made  without  committing  to  an  arc  going  in  the  opposite  direction.  For  example 
“Number  of  Offers  Received”  is  blacklisted  to  every  other  piece  of  evidence  in  the  model, 
because  competition  takes  place  only  after  the  other  factors  are  set  in  broad  terms. 

However,  as  will  be  seen  in  the  causal  model,  not  every  piece  of  evidence  that  can  arc  to 
“Number  of  Offers  Received”  does  so. 
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One-way  Blocks  in  Red,  Two-way  Blocks  in  Purple 


Figure  2.  Bayesian  Network  Blacklist 

In  a  smaller  number  of  cases,  the  blacklist  extends  to  arcs  in  both  directions.  This 
means  that  the  two  evidence  nodes,  connected  with  a  purple  arc  and  an  arrow  pointing  in 
both  directions  cannot  be  parents  nor  children  of  each  other.  In  the  case  of  IDV  and  linked, 
this  is  because  the  causal  link  is  difficult  adjudicate  because  of  confounding  factors.  For 
contracting  office  and  platform  pairings  that  have  IDVs  available,  the  number  of  preexisting 
interlinked  contracts  will  typically  be  higher.  However,  that  reflects  the  omitted  variable  of 
whether  there  are  active  IDVs  rather  than  a  direct  connection  between  the  two  evidence 
nods. 


Component  evidence  node  has  bidirectional  blacklisted  arcs  with  all  of  the 
contracting  approach  evidence  nodes  as  well  as  with  the  “Number  of  Offers  Received” 
outcome  node.  The  study  team  chose  to  block  these  links  because  the  literature  review  did 
not  find  a  theoretical  basis  for  the  organization  itself,  rather  than  the  characteristics  of  its 
contracts,  being  a  key  determiner  for  fixed-price  success  or  failure.  CSIS  is  separately 
examining  that  question  in  its  report  on  DoD  components,  which  is  also  being  presented  at 
this  conference.  Pending  the  outcome  of  that  study,  these  bidirectional  blacklisted  arcs  may 
be  revisited. 

Initial  Results 

After  the  iterated  whitelist  and  blacklist  generation  process,  Figure  3  shows  the 
ultimate  result.  In  the  figure,  the  blue  arrows  are  those  arcs  that  were  mandated  by 
whitelists.  With  the  exception  of  links  to  Duration  from  Ceiling,  IDV,  and  Comp,  the  direction 
of  the  remainder  of  the  arcs  was  locked  in  by  the  blacklist.  Thus,  PSR  did  not  have  to  have  a 
direct  connection  to  Duration,  but  the  alternative  was  disallowed. 

The  resulting  Bayesian  network  is  highly  interlinked,  as  is  shown  by  the  sheer 
number  of  arcs,  well  in  excess  of  the  small  number  required  by  the  white  list.  However,  there 
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are  two  notable  areas  where  connections  are  sparse.  Duration  is  influenced  by  six  different 
evidence  nodes  but  does  not  influence  Offr  or  any  other  node.  This  may  change  in  the  future 
as  a  result  of  introducing  greater  granularity  to  Duration  or  after  the  introduction  of  the  other 
dependent  variables.  The  second  evidence  node  that  is  remarkably  isolated  is  ’’Fixed  price 
or  Cost-base,”  which  suggests  humility  may  be  necessary  regarding  the  influence  of  the 
study  variable. 


Preliminary  Hypothesis  Testing 

This  initial  model  allows  for  preliminary  testing  of  four  of  the  five  study  hypotheses  as 
well  as  robustness  checking  using  five  different  controls.  The  study  team  developed  each  of 
these  hypotheses  and  controls  from  the  literature,  rather  than  through  learning  algorithm 
described  above. 

As  presently  configured,  the  various  evidence  nodes  are  not  granular  enough  to  fully 
test  the  hypotheses  proposed  in  the  early  stages  of  this  research.  However,  four  of  the  five 
hypotheses  can  be  tested.  These  hypotheses  were  often  formulated  with  a  greater  level  of 
specificity  than  the  model  presently  allows,  for  example,  examining  contracts  with  a  ceiling 
of  greater  than  $500  million  or  addressing  only  software  contracts.  Categories  which  only 
apply  to  a  small  fraction  of  contracts  can  increase  the  complexity  of  the  model  while 
reducing  its  statistical  power.  This  trade-off  can  be  worthwhile,  but  it  will  first  be  tested  with 
model  variants  and  only  when  most  insightful  will  these  additional  breakdowns  be  included 
in  the  model  as  a  whole. 
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The  current  Bayesian  model  enabled  five  different  controls  which  were  employed  by 
comparing  fixed  price  and  cost-based  Number  of  Offers  Received  for  a  different  subsets  of 
the  contract  dataset.  As  the  Bayesian  model  becomes  more  inclusive,  more  controls,  such 
as  separating  undefinitized  contract  actions,  will  be  added. 

•  Past  research  has  found  aircraft  and  drone  contracts  to  be  especially 
challenging,8  9  so  the  first  control  split  the  dataset  using  the  Platform  evidence 
node  (DoD,  2015;  Ritchie,  1997). 

•  Both  larger  and  smaller  contracts  sometimes  show  different  trends  than  those 
in  the  middle.  The  second  control  separated  out  large  contracts  as  those  with 
a  ceiling  of/over  $30  million.  The  third  control  separates  out  contracts  with  a 
ceiling  of  less  than  $1  million.10 

•  Indefinite  Delivery  Vehicles  are  associated  with  higher  rates  of  competition 
and  are  separated  out  for  the  fourth  control.* 11 

•  Finally  contracts  with  a  duration  of  greater  than  a  year  may  have  higher  risk 
thus  experience  different  dynamics. 

Hypothesis  1:  Large  R&D  contracts  will  perform  better  as  cost-based 
contracts. 

Our  interim  findings  generally  support  both  the  academic  literature12  and  policy 
documents13  that  posit  large  R&D  contracts  may  perform  better  under  cost-based  contracts. 
As  expected,  cost-based  contracts  had  a  substantially  lower  single  offer  competition  rate, 
22.5%  for  cost-based  versus  29.7%  for  fixed  price.  Cost-based  similarly  had  a  higher  rate  of 
competition  with  five  or  more  offer.  This  pattern  held  for  long  duration  contracts  and  for 
aircraft  contracts.  In  the  latter  case  fixed-price  contracts  received  only  a  single  offer  37.6% 


8  Aircraft  have  a  22  point  corresponding  effect  on  total  contract  growth  that  is  explained  by  schedule 
and  cost  overruns.  The  report  looked  at  time,  schedule,  and  differences  between  major  commands, 
but  not  contract  pricing  mechanism. 

9  During  WWII,  fixed-price  contracts  were  favored  for  aircraft,  but  due  to  factors  relating  to  the  “degree 
of  organization  by  weapons  companies,”  only  Britain  was  able  to  develop  primarily  fixed-price 
procedures. 

10  For  overall  DoD  contracting  in  201 3,  contracts  with  an  annual  value  of  less  than  $250,000  saw  the 
highest  rate  of  competition  with  two  or  more  offers  (62%),  and  that  rate  generally  declines  as  size  of 
contract  increases  (CSIS  Analysis  of  FPDS  Data).  That  breakpoint  did  not  align  with  the  buckets  of 
the  Ceiling  evidence  node  and  the  lower  than  $1  million  captured  this  dynamic  more  effectively  than 
lower  than  $100,000. 

11  Past  CSIS  research  has  found  that  contracts  using  a  purchase  orders  and  “Other  IDCs”  (Other 
IDCs  includes  Federal  Supply  Schedule,  Basic  Ordering  Agreements,  Blanket  Purchasing 
Agreements,  Government-Wide  Acquisition  Contracts)  have  notably  higher  rates  of  single-offer 
competition  than  other  vehicles  (CSIS,  Federal  Services  Contracting  and  the  Supporting  Industrial 
Base,  2000-2012).  Multi-award  IDVs  however  have  notably  higher  rates.  This  hypothesis  will  be 
retested  when  greater  granularity  in  the  vehicle  evidence  node  is  available. 

12  “In  particular,  we  find  that  the  [contracting  office]  principal  prefers  a  cost-plus  contract  in  cases  of 
large  R&D  projects  or  rising  innovation  benefits.  ...  The  agent  increases  its  research  out-lays  in 
response  to  a  higher  sharing  rate  when  the  expected  rewards  from  innovation  significantly  exceed 
research  costs”  (Goel,  1999). 

13  USD(AT&L)  describes  low  technical  risk  as  a  reason  to  choose  fixed-price  contracts.  Large  R&D 
contracts  are  known  for  their  technical  risk  (Kendall,  2015). 
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of  the  time  versus  1 1 .3%  for  cost-based!  In  these  categories  and  in  the  overall  sample, 
combination  contracts  more  closely  resembled  fixed-price  contracts  when  it  came  to  offers 
received.  Cost-plus  only  loses  its  advantage  for  IDV  contracts,  where  26%  of  contracts 
receive  only  a  single  offer  compared  to  24.9%  of  combination  contracts  and  20.5%  of  fixed- 
price  contracts.  This  discrepancy  merits  further  study  to  determine  what  sort  of  IDV  drives 
up  the  rate  of  single  offer  competition  for  cost-plus  contracts. 

Hypothesis  2:  Complex  projects,  as  measured  by  pre-Milestone  B  major 

defense  acquisition  project  status,  will  perform  better  as  cost-based  contracts. 

Complex  projects,  such  as  the  development  stage  of  Major  Defense  Acquisition 
Projects,  are  favored  for  cost  based  contracts  in  the  literature  (Bajari  &  Tadelis,  2001  )14  but 
not  supported  by  our  analysis  when  the  placeholder  of  linkages  was  used.  However,  the 
results  were  not  highly  robust  and  were  contradicted  for  two  of  the  five  controls.  The  single 
offer  competition  rate  for  cost-plus  contracts  was  six  percentage  points  lower  for  large 
contracts  and  two  percentage  points  lower  for  aircraft.  Since  both  large  contracts  and 
aircraft  and  drone  contracts  are  associated  with  MDAPs,  this  suggests  that  the  study  team 
will  need  to  refine  linkages  or  test  the  hypothesis  directly  by  looking  at  system  equipment 
codes. 


Hypothesis  3:  Contracts  with  a  longer  duration  will  perform  better  as  cost- 
based  contracts. 

While  prolonged  schedules  and  unforeseen  cost  growth  are  reasons  a  vendor  may 
prefer  cost-based  contracting,15  the  hypothesized  relationship  did  not  hold.  Unlike 
Hypothesis  2,  the  results  appeared  to  be  fairly  robust.  Overall,  25.3%  of  fixed-price  contracts 
with  durations  greater  than  a  year  receive  only  one  offer  compared  to  31 .4%  of  cost-plus 
contracts.  This  gap  narrows  to  less  than  a  percentage  point  for  large  contracts  and  aircraft 
and  drone  contracts, 

Combination  contract  rates  resembled  fixed-price  rates  in  most  categories,  with  the 
exception  of  aircraft  and  drones  where  more  than  half  of  all  competed  contracts  received 
only  one  offer.  Further  analysis  may  be  merited  to  determine  whether  the  hypothesis  holds 
when  the  threshold  is  set  at  two  or  more  years. 

Hypothesis  4:  The  potential  for  greater  competition  improves  fixed-price 
performance. 

The  hypothesis  that  fixed-price  contests  are  preferred  by  acquisition  officials  when 
they  are  likely  to  receive  more  competition16  may  indicate  that  contracts  that  are  more  likely 
to  be  competed  will  perform  better.  This  hypothesis  is  not  testable  with  this  dependent 
variable.  However,  given  the  high  variability  for  the  Number  of  Offers  Received  for  fixed- 
price  contracts  and  the  number  of  evidence  nodes  influencing  both  competition  and  number 
of  offers,  this  hypothesis  should  be  straightforward  to  test  in  future  stages. 


14  Cost-plus  contracts  are  preferred  to  fixed-price  contracts  when  a  project  is  more  complex. 

15  “But  price  redetermination  might  be  used  whenever  contingency  charges  otherwise  would  be 
included  in  a  contract  price  due  to  such  factors  as  prolonged  delivery  schedules,  unstable  market 
conditions  for  material  or  labor,  or  uncertainty  as  to  cost  of  performance”  (Fixed  Prices  and  Price 
Redetermination  in  Defense  Contracts). 

16  “[Government  principals]  would  prefer  a  fixed-price  contract  when  the  number  of  bidders  increases” 
(Goel,  2001). 
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Hypothesis  5:  Large  software  projects  perform  better  as  fixed-price  contracts. 

Our  results  surrounding  this  hypothesis  were  inconclusive.  We  based  the  hypothesis 
on  one  piece  of  literature  which  stated  that  vendors  prefer  fixed-price  contracts  for 
software,17  contradicting  other  literature  that  hypothesized  that  venders  prefer  cost-plus 
contracts  for  larger,  more  complicated  projects.  For  this  iteration,  Electronics  and 
Communications  services  were  used  as  a  proxy  for  software.  We  found  that  for  small 
contracts,  the  hypothesis  holds,  with  fixed-price  contracts  nearly  10  percentage  points  less 
likely  to  receive  one  offer  compared  to  cost-based  (28.8%  versus  38.7%).  However,  this 
relationship  reverses  itself  for  large  contracts;  29%  of  fixed-price  contracts  receive  only  a 
single  offer  versus  21 .2%  of  cost-based  contracts. 

Conclusions 

This  promising  hypothesis  testing  shows  the  value  of  the  Bayesian  network  built  by 
the  study  team.  The  analysis  of  single-offer  competition  for  larger  R&D  contracts  confirmed 
the  conventional  wisdom  that  vendors  would  be  less  likely  to  bid  on  fixed-price  contracts  but 
also  revealed  an  intriguing  wrinkle  that  this  does  not  hold  for  large  IDV  R&D  contracts.  In 
addition,  both  the  model  building  and  the  hypothesis  testing  surprisingly  failed  to  uncover  a 
relationship  between  contract  duration  and  the  number  of  offers  a  contract  received.  Both 
the  literature  and  practitioners  have  often  mentioned  the  importance  of  contract  duration, 
and  it  may  prove  more  consequential  for  contract  terminations  and  ceiling-raising  change 
orders  which  the  study  team  will  explore  as  this  work  continues. 

Finally,  the  study  team  is  excited  to  pursue  the  next  steps  of  refining  the  model  via 
external  consultation  and  also  expanding  the  model  to  include  the  remaining  two  dependent 
variables  as  well  as  a  small  number  of  intervening  variables.  The  latest  version  of  the  model 
will  be  available  at  the  CSISdefense  GitHub  account,  and  CSIS  welcomes  feedback  from 
readers.18 
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